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RETURN -TO- LI BC ATTACK BLOCKING SYSTEM AND METHOD 

BACKGROUND OF THE INVENTION 
Field Of The Invention 
5 The present invention relates to the protection of 

computer systems. More particularly, the present invention 
relates to a system and method of blocking Return- to-LIBC 
attacks . 

10 Description Of The Related Art 

Buffer overflow techniques have been used by malicious 
hackers and virus writers to attack computer systems. 
Buffers are data storage areas, which generally hold a 
predefined amount of finite data. A buffer overflow occurs 
15 when a program attempts to store data into the buffer, where 
the data is larger than the size of the buffer. 

One category of buffer overflow, sometimes called stack- 
based buffer overflow, involves overwriting stack memory, 
sometimes called the stack. Stack-based buffer overflow is 
20 typically caused by programs that do not verify the length of 
the data being copied into a buffer. 

When the data exceeds the size of the buffer, the extra 
data can overflow into the adjacent memory locations. In 
this manner, it is possible to corrupt valid data and 
25 possibly to change the execution flow and instructions. 

In the case of a Return- to-LIBC attack, the attacker 
overflows the stack in such a way that a return address will 
be replaced to point to a library function in a loaded 
library inside the process address space. Thus, when the 
30 return address is used by the overflowed process, a library 
function will be executed. This way the attacker runs at 
least one application programming interface (API) to run a 
command shell on the compromised system remotely. 



35 



Summary of the Invention 

A method includes stalling a call to a critical 
operating system (OS) function, looking up a value at the 
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previous top of stack, and determining whether the value is 
equivalent to an address of the critical OS function being 
called. If the value at the previous top of stack is 
equivalent to the address of the critical OS function being 
5 called, the method further includes taking protective action 
to protect a computer system. 

In one embodiment, if the value at the previous top of 
stack is equivalent to the address of the critical OS 
function being called, a determination is made that the 
10 critical OS function is being called via a RET (return) 

instruction as part of a Return- to-LIBC attack. By taking 
protective action such as terminating the critical OS 
function call, the Return- to-LIBC attack is blocked and 
defeated. 

15 Embodiments in accordance with the present invention are 

best understood by reference to the following detailed 
description when read in conjunction with the accompanying 
drawings . 

20 BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a diagram of a client - server system that 
includes a Return- to-LIBC attack blocking application 
executing on a host computer system in accordance with one 
embodiment of the present invention; 
25 FIG. 2 is a flow diagram of a host computer process in 

accordance with one embodiment of the present invention; 

FIG. 3 is a diagram of a hooked operating system 
function call flow in accordance with one embodiment of the 
present invention; 
30 FIG. 4 is a pseudocode representation of a stack at the 

instant when a critical OS function call is made in 
accordance with one embodiment of the present invention; 

FIG. 5 is a flow diagram of a critical OS function call 
from RET instruction check operation of the host computer 
35 process of FIG. 2 in accordance with one embodiment of the 
present invention; 
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FIG. 6 is a diagram of a hooked operating system 
function call flow in accordance with another embodiment of 
the present invention; and 

FIG. 7 is a pseudocode representation of an overflowed 
5 stack at the instant when a critical OS function call is made 
in accordance with another embodiment of the present 
invention . 

Common reference numerals are used throughout the 
drawings and detailed description to indicate like elements. 

10 

DETAILED DESCRIPTION 

In accordance with one embodiment, referring to FIGS. 6 
and 7, a method includes stalling a call 610 to a critical 
operating system (OS) function 308, looking up a value V at 
15 the previous top of stack at address [ESP-4] , and determining 
whether the value V is equivalent to an address of critical 
OS function 308. If the value V is equivalent to the address 
of critical OS function 308, the method further includes 
taking protective action such as terminating call 610. 
20 In one embodiment, if the value V is equivalent to the 

address of critical OS function 308, a determination is made 
that critical OS function 308 is being called via a RET 
(return) instruction 702 as part of a Return- to-LIBC attack. 
By taking protective action such as terminating call 610, the 
25 Return- to-LIBC attack is blocked and defeated. 

More particularly, FIG. 1 is a diagram of a client- 
server system 100 that includes a Return- to-LIBC attack 
blocking application 106 executing on a host computer system 
102, e.g., a first computer system, in accordance with one 
30 embodiment of the present invention. 

Host computer system 102, sometimes called a client or 
user device, typically includes a central processing 
unit (CPU) 108, hereinafter processor 108, an input output 
(I/O) interface 110, and a memory 114. In one embodiment, 
35 memory 114 includes a page based virtual memory system that 
uses pages, e.g., memory areas. 
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For example, Windows® NT and Windows® 2000 are 32 -bit 
operating systems widely used on home and business computer 
systems. Windows® NT and Windows® 2000 provide page-based 
virtual memory management schemes that permit programs to 
realize a 4GB (gigabyte) virtual memory address space. In 
one embodiment, when processor 108 is running in virtual 
memory mode, all addresses are assumed to be virtual 
addresses and are translated, or mapped, to physical 
addresses each time processor 108 executes a new instruction 
to access memory. 

Conventionally, the 4GB virtual memory address space is 
divided into two parts: a lower 2GB user address space, also 
referred to as user mode address space or ring 3, available 
for use by a program; and, a high 2GB system address space, 
also referred to as kernel address space or ring 0, reserved 
for use by the operating system. 

To protect the integrity of the operating system code 
and other kernel address space code and data structures from 
errant or malicious programs and to provide efficient system 
security (user rights management) , Windows® NT and Windows® 
2000 separate code executing in the user address space, e.g., 
user mode, from code executing in the kernel address space, 
e.g., kernel mode. User mode code typically does not have 
direct access to kernel mode code and has restricted access 
to computer system resources and hardware. To utilize kernel 
mode code functionalities, such as access to disk drives and 
network connections, user mode programs utilize system calls, 
sometimes called operating system (OS) function calls or 
APIs, that interface between the user mode and kernel mode 
functions . 

Host computer system 102 may further include standard 
devices like a keyboard 116, a mouse 118, a printer 120, and 
a display device 122, as well as, one or more standard 
input/output (I/O) devices 123, such as a compact disk (CD) 
or DVD drive, floppy disk drive, or other digital or waveform 
port for inputting data to and output ting data from host 
computer system 102. In one embodiment, Return- to-LIBC 
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attack blocking application 106 is loaded into host computer 
system 102 via I/O device 123, such as from a CD, DVD or 
floppy disk containing Return- to-LIBC attack blocking 
application 106. 

5 Host computer system 102 is coupled to a server system 

130 of client-server system 100 by a network 124. Server 
system 130 typically includes a display device 132, a 
processor 134, a memory 136, and a network interface 138. 

Further, host computer system 102 is also coupled to a 

10 hacker computer system 104 of cl ient - server system 100 by 

network 124. In one embodiment, hacker computer system 104 
is similar to host computer system 102, for example, includes 
a central processing unit, an input output (I/O) interface, 
and a memory. Hacker computer system 104 may further include 

15 standard devices like a keyboard, a mouse, a printer, a 

display device and an I/O device (s) . The various hardware 
components of hacker computer system 104 are not illustrated 
to avoid detracting from the principals of the invention. 

Network 124 can be any network or network system that is 

20 of interest to a user. In various embodiments, network 

interface 138 and I/O interface 110 include analog modems, 
digital modems, or a network interface card. 

Return-to-LIBC attack blocking application 106 is stored 
in memory 114 of host computer system 102 and executed oh 

25 host computer system 102. The particular type of and 

configuration of host computer system 102, hacker computer 
system 104, and server system 130 are not essential to this 
embodiment of the present invention. 

FIG. 2 is a flow diagram of a host computer process 200 

30 in accordance with one embodiment of the present invention. 

Referring now to FIGS. 1 and 2 together, execution of Return- 
to-LIBC attack blocking application 106 by processor 108 
results in the operations of host computer process 200 as 
described below in one embodiment . 

35 From an enter operation 202, flow moves to a hook 

critical operating system (OS) function (s) operation 204. In 
hook critical OS function (s) operation 204, the critical 
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operating system functions, e.g., at least one critical 
operating system function, of host computer system 102 are 
hooked. In one embodiment, a system level, e.g., a kernel 
mode module or kernel mode driver, hooks the critical 
5 operating system functions. Further, in one embodiment, an 
operating system function is hooked by redirecting calls to 
the operating system function, for example, to a hook module 
in accordance with the present invention. 

In one embodiment, an operating system function is 
10 critical if it is necessary for a first application, e.g., a 
parent application, to cause execution of a second 
application, e.g., a child application. In one particular 
embodiment, an operating system function is critical if it is 
necessary or likely to be used by a malicious parent 
15 application, e.g., an application which contains or uses 
malicious code, e.g., located on the stack, to execute a 
child application, where the child application allows remote 
access, e.g., remote system level access. Examples of child 
applications include the command prompt or "cmd.exe" on a 
20 Windows® operating system and "/bin/sh" on a UNIX or UNIX 
like, e.g., FreeBSD or MacOS x, operating system. As used 
herein, a child application is not dependent upon a parent 
application, i.e., once the child application is executed the 
parent application can be terminated without termination of 
25 the child application. 

In one embodiment, on a Windows® operating system, the 
CreateProcessO and System () are critical OS functions. In 
yet another embodiment, critical OS functions are located in 
the C- library hence the name "Return- to-LIBC" attack. 
30 As is well known to those of skill in the art, system 

calls expose all kernel functionality that user-mode programs 
require. User-mode programs need to utilize the 
functionality provided by the kernel, for example, to access 
disk drives, network connections, and shared memory. More 
35 particularly, because the processor prevents direct access to 
kernel mode functions by user-mode programs, user-mode 
programs use system calls, which form the only permitted 
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interface between user-mode and kernel mode. In accordance 
with one embodiment, system calls include critical OS 
function calls and non-critical OS function calls. 

From hook critical OS function (s) operation 204, flow 
5 moves to a call to critical OS function operation 206. In 

call to critical OS function operation 206, a call, sometimes 
called a critical OS function call, to a critical OS function 
is made by a call module of a parent application. The parent 
application may be malicious or non-malicious. More 
10 particularly, a critical OS function call is made by a call 
module of a parent application to an OS function that was 
hooked in hook critical OS function (s) operation 204. 

In accordance with one embodiment of the present 
invention, a call module includes the critical OS function 
15 call instruction (s) , i.e., the instruction or set of 

instructions that originates the critical OS function call. 
The call module may be malicious or non-malicious. The 
parent application includes the call module, or, in one 
embodiment, the parent application is the call module. 
20 For example, system calls are originated by execution of 

a CALL instruction or a RET instruction (in the case of a 
Return-to-LIBC attack) . In one embodiment, a CALL 
instruction is used to divert execution to a particular 
function. A RET instruction, sometimes called a return 
25 instruction, is used to return from a subroutine or marks the 
end of a main function. 

From call to critical OS function operation 206, flow 
moves to a stall call operation 208. In stall call operation 
208, the critical OS function call of operation 206 to the 
30 critical OS function is stalled, i.e., is prevented from 

reaching the operating system. By stalling the critical OS 
function call, execution of the critical OS function is 
stalled. 

From stall call operation 208, flow moves to a critical 
35 OS function call from RET instruction check operation 210. 

In check operation 210, a determination is made as to whether 
the critical OS function call originated from execution of a 
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RET instruction. If a determination is made in check 
operation 210 that the critical OS function call did not 
originate from execution of a RET instruction, flow moves to 
an allow call to proceed operation 212. 
5 In allow call to proceed operation 212, the critical OS 

function call is allowed to proceed. More particularly, the 
critical OS function call is passed to the operating system. 
As discussed above, the critical OS function call was stalled 
in stall call operation 208. From allow call to proceed 
10 operation 212, flow moves to and exits at an exit operation 
214 or waits for the next critical OS function call and 
returns to operation 206. 

In a typical Return- to-LIBC attack, a return address is 
replaced with a malicious return address pointing to a 
15 library function in a loaded library, e.g., the C-library, 
inside the process address space by exploiting a buffer 
overflow. Thus, when the malicious return address is used by 
the RET instruction of the call module, a critical OS 
function call is made. Thus, because a determination is made 
20 that the critical OS function call did not originate from 
execution of a RET instruction in check operation 210, the 
likelihood that the call module is malicious code is minimal. 
In one embodiment, malicious code is defined as any computer 
program, module, set of modules, or code that enters a 
25 computer system without an authorized user's knowledge and/or 
without an authorized user's consent. 

However, if a determination is made in check operation 
210 that the critical OS function call did originate from 
execution of a RET instruction, flow moves, optionally, to a 
30 known false positive check operation 216 (or directly to a 

take protective action operation 218 if known false positive 
check operation 216 is not performed) . 

In known false positive check operation 216, a 
determination is made as to whether the critical OS function 
35 call is a known false positive. A known false positive 

critical OS function call is a critical OS function call that 
originates from execution of a RET instruction but is, in 
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fact, safe, i.e., is not associated with malicious code. An 
example of a known false positive critical OS function call 
is the case when non malicious code pushes an OS function 
address on to the stack and then transfers control to the OS 
5 function via a RET instruction. Illustratively, a user- 
defined or downloadable exclusion and/or inclusion list is 
used to determine whether the critical OS function call is a 
known false positive. 

If a determination is made in check operation 216 that 
10 the critical OS function call is a known false positive 
critical OS function call, flow moves to allow call to 
proceed operation 212, which is performed as discussed above. 
Conversely, if a determination is made in check operation 216 
that the critical OS function call is not a known false 
15 positive critical OS function call, flow moves to a take 
protective action operation 218. 

In take protective action operation 218, protective 
action is taken to prevent the malicious code of or used by 
the call module from causing damage to or exploiting host 
20 computer system 102. For example, the critical OS function 
call is terminated. More particularly, the critical OS 
function call is not passed to the operating system but is 
terminated. As discussed above, the critical OS function 
call was stalled in stall call operation 208. 
25 By terminating the critical OS function call, the 

malicious code of the call module is prevented from 
exploiting and/or damaging host computer system 102. In one 
embodiment, by terminating the critical OS function call, the 
child application is prevented from being executed. By 
30 preventing execution of the child application, remote access 
is denied thus preventing unauthorized access by malicious 
hackers as well as by replicating malware, e.g., worms. 

In one embodiment, because a determination is made in 
check operation 210 that the critical OS function call did 
35 originate from execution of a RET instruction, the likelihood 
that the call module is malicious code is significant. 
However, by terminating the critical OS function call, the 



SYMC1038 



critical OS function is prevented from being execution. By 
preventing execution of the critical OS function, remote 
access is denied, thus preventing unauthorized access by 
malicious hackers and also by replicating malware, e.g., 
worms . 

As another example of protective action, the parent 
application including the call module and/or a malicious 
thread running within the context of the parent application 
is terminated. Termination of applications is well known to 
those of skill in the art and so is not discussed further for 
clarity of discussion. 

Flow moves from take protective action operation 218, 
optionally, to a notify host computer system 
user/administrator operation 220 (or directly to exit 
operation 214 if operation 220 is not performed) . In notify 
host computer system user/administrator operation 22 0, the 
user of host computer system 102 and/or the administrator are 
notified that protective action has been taken on host 
computer system 102, e.g., that a call, a parent application 
and/or a call module have been terminated. The user and/or 
administrator can be notified using any one of a number of 
techniques, e.g., by using a pop up window, by writing to a 
file and/or otherwise by logging the event. Further, a 
notification can be provided to a security center. 

From notify host computer system user/administrator 
operation 220, flow moves to and exits at exit operation 214 
or waits for the next critical OS function call and returns 
to operation 206. 

FIG. 3 is a diagram of a hooked operating system 
function call flow 300 in accordance with one embodiment of 
the present invention. Referring now to FIGS. 2 and 3 
together, a hook module 306 is used to hook calls to a 
critical OS function 308. In one embodiment, hook critical 
OS function (s) operation 204 is implemented using hook module 
306, which is part of Return- to-LIBC blocking application 106 
(FIG. 1) . 
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More particularly, a hooked system service table 304 
routes noncritical OS function calls directly to the 
operating system (not shown) . However, hooked system service 
table 304 routes critical OS function calls to hook module 
5 306, e.g., a kernel mode module or kernel mode driver. 

As is well known to those of skill in the art, a system 
service table, sometimes called a dispatch table or a system 
call table, relates system calls to specific addresses within 
the operating system kernel. Hooked system service table 304 
10 in accordance with one embodiment of the present invention, 
redirects critical OS function calls to hook module 306 and 
from the specific addresses within the operating system 
kernel to which the critical OS function calls would 
otherwise be directed. 
15 Although FIG. 3 describes one example of a hooked 

operating system function call path, in light of this 
disclosure, those of skill in the art will understand that 
other techniques can be used to hook operating system 
function(s). The particular technique used depends, for 
20 example, on the particular operating system. 

In one embodiment, hook module 306 is used to stall a 
critical OS function call during stall call operation 208 of 
FIG. 2. Further, hook module 306 continues to stall the 
critical OS function call during critical OS function call 
25 from RET instruction check operation 210 (and known false 

positive check operation 216 if performed) . Hook module 306 
allows the critical OS function call to proceed to the 
operating system and thus to critical OS function 308 during 
allow call to proceed operation 212. Conversely, hook module 
30 306 terminates the critical OS function call and/or takes 
other protective action during take protective action 
operation 218. 

In accordance with this embodiment, a critical OS 
function call 310 originates from a call module 302 during 
35 call to critical OS function operation 206. Critical OS 

function call 310 is routed by hooked system service table 
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304 to hook module 306. Critical OS function call 310 is 
stalled by hook module 306 in stall call operation 208. 

FIG. 4 is a pseudocode representation of a stack 440 at 
the instant when critical OS function call 310 is made in 
5 accordance with one embodiment of the present invention. In 
one embodiment, stack 440 is part of memory 114 (FIG. 1). 
Illustratively, stack 440 is an area of dwords (32 bit data 
areas) in memory 114 which applications use to store data 
temporarily. 

10 The register ESP, i.e., the extended stack pointer, 

holds the top of the stack. More particularly, the register 
ESP holds a value, sometimes called the stack pointer, that 
is the address of the top of the stack. Accordingly, the 
address of the top of the stack is sometimes called address 

15 [ESP] . The top of the stack is the address where 

instructions that use the stack, e.g., PUSH, POP, CALL and 
RET instructions, actually use the stack. The operation and 
use of stacks, stack pointers, and the top of the stack are 
well known to those of skill in the art. Note that the 

20 bottom of the stack is the uppermost area of stack 440 in the 
view of FIG. 4, i.e., the top of stack 440 is down in the 
view of FIG. 4 and the bottom of stack 440 is up in the view 
of FIG. 4. Generally, the addresses in stack 440 decrease 
towards the top of the stack, i.e., the addresses decrease 

25 going down in the view of FIG. 4. 

Referring now to FIGS. 3 and 4 together, to originate 
critical OS function call 310, call module 302 executes 
instructions 402, 404, 406, and 408. More particularly, call 
module 302 includes instructions 402, 404 and 406 that push 

30 parameter P2 , parameter PI, and parameter P0, respectively, 
on to stack 440. Call module 302 further includes an 
instruction 408 that calls a critical OS function and an 
instruction 410 that decrements the stack pointer for cleanup 
of stack 440 as those of skill in the art will understand. 

35 Execution of instructions 402, 404 and 406 push 

parameter P2 , parameter PI, and parameter P0, respectively, 
on to stack 440 as shown by the arrows. Execution of 
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instruction 408 causes critical OS function call 310 to be 
made and pushes the return address RA to which the critical 
OS function is to return, e.g., the address of instruction 
410, on to stack 440 as shown by the arrow. 

Accordingly, when critical OS function call 310 is made 
by call module 302, the top of the stack at address [ESP] 
contains the return address RA. The previous top of stack, 
sometimes called the previous DWORD, contains a value V 
typically unrelated to critical OS function call 310. In one 
embodiment, the previous top of stack is the memory address 
at which the last (most recent) value from the stack was 
popped, i.e., read. The value, sometimes called data, in the 
previous top of the stack is still there because the value is 
just read from the stack when it is popped and not 
overwritten. In one embodiment, the previous top of the 
stack is at address [ESP-4] , i.e., the stack pointer minus 4 
bytes . 

FIG. 5 is a flow diagram of critical OS function call 
from RET instruction check operation 210 of host computer 
process 2 00 of FIG. 2 in accordance with one embodiment of 
the present invention. Referring now to FIGS. 2, 3, 4 and 5 
together, from an enter operation 502 (and from stall call 
operation 208 of FIG. 2), flow moves to a look up value at 
previous top of stack operation 504. In look up value at 
previous top of stack operation 504, the value V at the 
previous top of stack at address [ESP-4] is determined, e.g., 
read. Reading values from stack addresses is well known to 
those of skill in the art. From look up value at previous 
top of stack operation 504, flow moves to a value equivalent 
to address of critical OS function check operation 506. 

In check operation 506, a determination is made as to 
whether the value V at the previous top of stack at address 
[ESP-4] determined in operation 504 is equivalent, and in one 
embodiment, exactly equal, to the address of the critical OS 
function being called. In accordance with this embodiment, 
critical OS function call 310 is a call to critical OS 
function 3 08, which is located at a particular address in 
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memory 114 (FIG. 1). The particular address of critical OS 
function 308 is obtained using any one of a number of 
techniques well known to those of skill in the art and the 
particular technique used is not essential to the present 
5 invention. 

As discussed above, the previous top of stack at address 
[ESP-4] contains a value V typically unrelated to critical OS 
function call 310 and, specifically, unrelated to the address 
of critical OS function 308. Accordingly, in this embodiment 
0 where call module 3 02 is not malicious, a determination is 
made in check operation 506 that the value at the previous 
top of stack at address [ESP-4] is not equivalent to the 
address of critical OS function 308. Accordingly, flow moves 
to allow call to proceed operation 212 through operation 508. 
5 FIG. 6 is a diagram of a hooked operating system 

function call flow 300A in accordance with another embodiment 
of the present invention. Referring now to FIGS. 2 and 6, 
hook module 306 is used to hook calls to critical OS function 
308 routed from hooked system service table 304 as discussed 
20 above in reference to FIG. 3, the discussion of which is 
herein incorporated by reference. 

In accordance with this embodiment, a critical OS 
function call 610 originates from a malicious call module 602 
during call to critical OS function operation 206. For 
25 example, control is transferred from a parent application 612 
to malicious call module 602. As discussed further below, 
instead of returning control to parent application 612, 
malicious call module 602 originates critical OS function 
call 610 by executing a RET instruction using a malicious 
30 return address placed on to a stack as a result of a buffer 
overflow in a typical Return- to-LIBC attack. 

Critical OS function call 610 is routed by hooked system 
service table 304 to hook module 306. Critical OS function 
call 610 is stalled by hook module 306 in stall call 
35 operation 2 08. 

FIG. 7 is a pseudocode representation of an overflowed 
stack 440A at the instant when critical OS function call 610 
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is made in accordance with another embodiment of the present 
invention. Referring now to FIGS. 6 and 7 together, to 
originate critical OS function call 610, malicious call 
module 602 executes a RET instruction 702. 
5 More particularly, malicious call module 602 includes 

RET instruction 702 (FIG. 7) that is used to return control 
to parent application 612 during normal operation. However, 
by exploiting a buffer overflow, malicious parameters are 
loaded into overflowed stack 44 OA. For example, a buffer 
10 overflow is exploited from hacker computer system 104 (FIG. 
1) . 

Specifically, a buffer overflow is exploited to load 
parameters PZ, PY, PX and a dummy return address DRA into 
addresses [ESP+12] , [ESP+8] , [ESP+4] , [ESP] , respectively, of 
15 overflowed stack 440A. Also, the value V, sometimes called a 
malicious return address, is loaded into address [ESP-4] of 
overflowed stack 440A during the exploitation of the buffer 
overflow. 

Accordingly, during execution of RET instrxiction 702 of 

20 malicious call module 602, the value V is read, sometimes 
called popped, from overflowed stack 440A. Accordingly, 
malicious call module 602 returns to the address specified by 
value V. As part of the Return- to-LIBC attack, the value V 
is the address of critical OS function 308. Thus, malicious 

25 call module 602 attempts to transfer control to critical OS 
function 308 by generating critical OS function call 610. 

Accordingly, when critical OS function call 610 is made 
by malicious call module 602, the previous top of stack at 
address [ESP-4] contains a value V, which is the address of 

30 critical OS function 308. Further, the top of the stack at 
address [ESP] contains dummy return address DRA followed by 
PX, PY and PZ, which is what critical OS function 308 would 
expect to see in the stack and thus intentionally placed 
there during the Return- to-LIBC attack. 

35 Referring now to FIGS. 2, 5, 6 and 7 together, from 

enter operation 502 (and from stall call operation 208 of 
FIG. 2), flow moves to look up value at previous top of stack 
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operation 504. In operation 504, the value V at the previous 
top of stack at address [ESP-4] , i.e., the address of 
critical OS function 308, is determined. From operation 504, 
flow moves to value equivalent to address of critical OS 
function check operation 506. 

In check operation 506, a determination is made that the 
value V at the previous top of stack at address [ESP-4] 
determined in operation 504 is equivalent to the address of 
the critical OS function being called. 

Accordingly, in this embodiment where malicious call 
module 602 is malicious, a determination is made in check 
operation 506 that the value at the previous top of stack at 
address [ESP-4] is equivalent to the address of critical OS 
function 308. Accordingly, flow moves to known false 
positive check operation 216 (or directly to take protective 
action operation 218 if known false positive check operation 
216 is not performed) through operation 510. 

After a determination that critical OS function call 610 
is not a known false positive in check operation 216, 
protective action is taken in take protective action 
operation 218. 

Illustratively, hook module 306 terminates critical OS 
function call 610. By terminating critical OS function call 
610, execution of critical OS function 308 is prevented. 
This, in turn, denies remote access thus preventing 
unauthorized access to host computer system 102 by malicious 
hackers and also by replicating malware, e.g., worms. 

Optionally, operation 220 is performed as discussed 
above and flow exits at exit operation 214. 

Referring again to FIG. 1, Return- to-LIBC attack 
blocking application 106 is in computer memory 114. As used 
herein, a computer memory refers to a volatile memory, a non- 
volatile memory, or a combination of the two. 

Although Return-to-LIBC attack blocking application 106 
is referred to as an application, this is illustrative only. 
Return-to-LIBC attack blocking application 106 should be 
capable of being called from an application or the operating 
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system. In one embodiment, an application is generally- 
defined to be any executable code. Moreover, those of skill 
in the art will understand that when it is said that an 
application or an operation takes some action, the action is 
5 the result of executing one or more instructions by a 

processor. In one embodiment, Return- to-LIBC attack blocking 
application 106 is implemented as a system level, e.g., 
kernel mode driver. 

While embodiments in accordance with the present 

10 invention have been described for a client-server 

configuration, an embodiment of the present invention may be 
carried out using any suitable hardware configuration 
involving a personal computer, a workstation, a portable 
device, or a network of computer devices. Other network 

15 configurations other than client-server configurations, e.g., 
peer-to-peer, web-based, intranet, internet network 
configurations, are used in other embodiments. 

Herein, a computer program product comprises a medium 
configured to store or transport computer readable code in 

20 accordance with an embodiment of the present invention. Some 
examples of computer program products are CD-ROM discs, DVDs, 
ROM cards, floppy discs, magnetic tapes, computer hard 
drives, servers on a network and signals transmitted over a 
network representing computer readable code. 

25 As illustrated in FIG. 1, this medium may belong to the 

computer system itself. However, the medium also may be 
removed from the computer system. For example, Return- to- 
LIBC attack blocking application 106 may be stored in 
memory 136 that is physically located in a location different 

30 from processor 108. Processor 108 should be coupled to the 
memory 136. This could be accomplished in a cl ient - server 
system, or alternatively via a connection to another computer 
via modems and analog lines, or digital interfaces and a 
digital carrier line. 

35 More specifically, in one embodiment, host computer 

system 102 and/or server system 130 is a portable computer, a 
workstation, a two-way pager, a cellular telephone, a digital 
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wireless telephone, a personal digital assistant, a server 
computer, an Internet appliance, or any other device that 
includes components that can execute the Return- to-LIBC 
attack blocking functionality in accordance with at least one 
of the embodiments as described herein. Similarly, in 
another embodiment, host computer system 102 and/or server 
system 130 is comprised of multiple different computers, 
wireless devices, cellular telephones, digital telephones, 
two-way pagers, or personal digital assistants, server 
computers, or any desired combination of these devices that 
are interconnected to perform, the methods as described 
herein . 

In view of this disclosure, the Return- to-LIBC attack 
blocking functionality in accordance with one embodiment of 
present invention can be implemented in a wide variety of 
computer system configurations. In addition, the Return-to- 
LIBC attack blocking functionality could be stored as 
different modules in memories of different devices. For 
example, Return-to-LIBC attack blocking application 106 could 
20 initially be stored in server system 130, and then as 
necessary, a portion of Return-to-LIBC attack blocking 
application 106 could be transferred to host computer 
system 102 and executed on host computer system 102. 
Consequently, part of the Return-to-LIBC attack blocking 
functionality would be executed on processor 134 of server 
system 13 0, and another part would be executed on 
processor 108 of host computer system 102. In view of this 
disclosure, those of skill in the art can implement various 
embodiments of the present invention in a wide-variety of 
physical hardware configurations using an operating system 
and computer programming language of interest to the user. 

In yet another embodiment, Return-to-LIBC attack 
blocking application 106 is stored in memory 13 6 of server 
system 130. Return-to-LIBC attack blocking application 106 
is transferred over network 124 to memory 114 in host 
computer system 102. In this embodiment, network 
interface 138 and I/O interface 110 would include analog 



25 
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modems, digital modems, or a network interface card. If 
modems are used, network 124 includes a communications 
network, and Return- to-LIBC attack blocking application 106 
is downloaded via the communications network. 

This disclosure provides exemplary embodiments of the 
present invention. The scope of the present invention is n 
limited by these exemplary embodiments. Numerous variation 
whether explicitly provided for by the specification or 
implied by the specification or not, may be implemented by 
one of skill in the art in view of this disclosure. 
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